Location-Aware Content and Location-Based Advertising with a Mobile Device

ABSTRACT

Many mobile devices such as smart phones can generate content ranging from text and audio, to images and videos that are taken with their cameras. These devices are also equipped with wireless radios and magnetometers that can be used to obtain position and orientation information. It is possible to allow the user of the mobile phone to add the position and orientation information of the mobile and date/time to the content that the mobile generates. Thus, emails, Short Message Service (SMS) messages, blog entries, audio recordings, and pictures/video that are generated with the mobile can be geotagged with the mobile&#39;s position and orientation, as well as the date and time. This information can then be used by a location-based contextual advertising server to provide more relevant on-line ads.

CLAIM OF BENEFIT TO RELATED APPLICATIONS

The present application claims the benefit of U.S. Provisional PatentApplication 61/231,902, entitled, “Location-Aware Content andLocation-Based Advertising with a Mobile Device,” filed Aug. 6, 2009.The contents of U.S. Provisional application 61/231,902 are herebyincorporated by reference.

BACKGROUND

Many web advertising sites use user profile information such as sex,age, marital status, education, and income to target online adverting tousers. Some sites offer contextual advertising, where the ads are textor links that are relevant to the page that the user is accessing. Forexample, Gmail™ indexes email messages, and based on the message contentadds text and links to related pages. Location-based contextualadvertising can take contextual advertising one step further bytargeting the ads to both context and the user's location.

Mobile devices such as smart-phones are information and communicationdevices that can generate text as well as multimedia content such aspictures, audio and video. There are situations when it is useful toknow the exact time and location of the mobile when the content wasgenerated. For instance, when users add pictures and video to theironline albums, social networking sites (e.g. Facebook®, MySpace™,YouTube®, Twitter™ and LinkedIn®), and other web sites they manuallydescribe and label the data, time and location of the content. Likewise,servers that handle email, Short Message Service (SMS), and blog entriesnote the time they received the content, rather than the time thecontent was created by a client device. Geotagging, which is the processof adding geographical identification data to multimedia content, solvesthese problems by adding self-describing metadata to the content. Somephoto sharing web sites support geotagging.

Other systems have used positioning technologies to providelocation-based information and advertising. For instance, a wirelessnetwork with base stations has been used to locate a subscriber and usethe location to filter responses to information requests. Also, cellularbase stations have been used to locate a cellular phone user and providelocation-specific information to travelers such as airline and trafficinformation. Other systems have utilized GPS to locate a user andprovide advertising that takes into account the user's location andprofile. A location-based content targeted online advertising systemhave been described where one of the factors that determines the contentof the advertisement is the location of the user as determined by alocation server. Others have described a location-based advertisingsystem that uses a GPS-based polling method to locate the user.

BRIEF SUMMARY

Some embodiments provide a method and system for utilizing geotagginginformation of user content to locate the user. Geotagging informationof user content is used in some embodiments to enable location-basedcontextual advertising. Some embodiments use, in part, the geotagging ofthe multimedia content of mobile users to determine their location andprovide location-based contextual advertising.

In some embodiments, a mobile device creates content for a user.Depending on the user's security settings, the mobile device's positionand orientation as well as date and time are tagged to some or all ofthe created content. The mobile device then transmits the content to aserver for storing the content in one or more databases. The mobiledevice then receives location-based advertising from one or moreservers.

Some embodiments provide a method of providing location-basedadvertising to a mobile device. The method receives content and metadataassociated with the content from the mobile device. The metadataincludes a geographical identification of the content. The method sendslocation-based information to the mobile device based on thegeographical identification of the content. In some embodiments thegeographical identification includes latitude and longitude coordinatesof a location where the content was generated. In some embodiments, thegeographical identification includes an orientation of the mobile devicewhen the content was generated, wherein sending location-basedinformation comprises sending information to the mobile device based onthe orientation of the mobile device.

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the invention. It is not meant to be an introductionor overview of all inventive subject matter disclosed in this document.The Detailed Description that follows and the Drawings that are referredto in the Detailed Description will further describe the embodimentsdescribed in the Summary as well as other embodiments. Accordingly, tounderstand all the embodiments described by this document, a full reviewof the Summary, Detailed Description and the Drawings is needed.Moreover, the claimed subject matters are not to be limited by theillustrative details in the Summary, Detailed Description and theDrawing, but rather are to be defined by the appended claims, becausethe claimed subject matters can be embodied in other specific formswithout departing from the spirit of the subject matters.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 conceptually illustrates a geographic area in some embodimentsthat has multiple wireless networks and a wireless mobile device thathas one or more wireless transceivers that use such networks tocommunicate information and find its location.

FIG. 2 conceptually illustrates a wireless mobile device in someembodiments that creates multimedia content, tags the content withposition/orientation/timestamp information, and uploads the content to aset of location-based application servers which in turn updates contentdatabases and user databases.

FIG. 3 conceptually illustrates a process for tagging and storingcontent generated by a mobile device in some embodiments.

FIG. 4 conceptually illustrates an example of a graphical user interfacefor setting security preferences in some embodiments.

FIG. 5 conceptually illustrates a system of some embodiments that usesgeotag and timestamp information from user-provided content to determineuser location and provide contextual location-based advertising.

FIG. 6 conceptually illustrates a flow diagram for providinglocation-based contextual advertising in some embodiments.

FIG. 7 conceptually illustrates a process performed by a mobile deviceto send content to one or more servers and receive location-basedinformation in some embodiments.

FIG. 8 conceptually illustrates a process performed by one or moreservers in some embodiments to receive content from a mobile device andprovide the information to one or more authorized servers.

FIG. 9 conceptually illustrates a process performed by one or moreservers in some embodiments to receive content from a mobile device andprovide location-based information to the same or different clients.

FIG. 10 conceptually illustrates a process performed by one or moreservers to provide location information of a mobile device to arequesting party in some embodiments.

FIG. 11 conceptually illustrates an example interface of someembodiments for a user that uses a mobile device to connect to anetworked server and access some of the information in the databases.

FIGS. 12 and 13 conceptually illustrate examples of location-basedadvertising for email in some embodiments.

FIG. 14 conceptually illustrates a process performed by one or moreservers to provide location-based advertisement in some embodiments.

FIG. 15 conceptually illustrates examples of location-based SMS messagesand location-based advertising with links to directions, VoIP calling,user ratings, advertiser's web site, and advertiser's coupons in someembodiments.

FIG. 16 conceptually illustrates a computer system with which someembodiments of the invention are implemented.

DETAILED DESCRIPTION

In the following detailed description of the invention, numerousdetails, examples, and embodiments of the invention are set forth anddescribed. However, it will be clear and apparent to one skilled in theart that the invention is not limited to the embodiments set forth andthat the invention may be practiced without some of the specific detailsand examples discussed.

Some embodiments provide a method and system for utilizing geotagginginformation of user content to locate the user. Geotagging informationof user content is used in some embodiments to enable location-basedcontextual advertising. Some embodiments use, in part, the geotagging ofthe multimedia content of mobile users to determine their location andprovide location-based contextual advertising.

In some embodiments, a mobile device creates content for a user.Depending on the user's security settings, the mobile device's positionand orientation as well as date and time are tagged to some or all ofthe created content. The mobile device then transmits the content to aserver for storing the content in one or more databases. The mobiledevice then receives location-based advertising from one or moreservers.

Some embodiments provide a method of providing location-basedadvertising to a mobile device. The method receives content and metadataassociated with the content from the mobile device. The metadataincludes a geographical identification of the content. The method sendslocation-based information to the mobile device based on thegeographical identification of the content. In some embodiments thegeographical identification includes latitude and longitude coordinatesof a location where the content was generated. In some embodiments, thegeographical identification includes an orientation of the mobile devicewhen the content was generated, wherein sending location-basedinformation comprises sending information to the mobile device based onthe orientation of the mobile device.

In some embodiments, the metadata further includes a time when thecontent was generated and the method sends time sensitive information tothe mobile device. When a difference between the time when the contentwas generated and a current time is more than a threshold, the method insome embodiments (1) determines a velocity of the mobile device based ona set of previously received content, (2) determines a direction of themobile device based on the set of previously received content, and (3)interpolates (or extrapolates) the location of the mobile device basedon the location of the generated content and the velocity and thedirection of the mobile device.

In some embodiments, the method determines a context from the contentreceived from the mobile device and narrows the location-basedinformation based on the determined context. The location-basedinformation includes location-based advertising and/or informationrequested by the mobile device.

In some embodiments, the mobile device does not transmit live trackingof the mobile device location. In some embodiments live tracking is notdone because the mobile device does not support live tracking. In otherembodiments, the user has set security preferences such that livetracking is disabled.

In some embodiments, the method sends a telephone link to the mobiledevice to contact a service provider associated with the location-basedinformation. The telephone link in some embodiments is a link to a Voiceover Internet Protocol (VoIP) server to make a call. In someembodiments, the method provides a map associated with thelocation-based information to the mobile device.

Several more detailed embodiments of the invention are described insections below. Section I describes use of geotagging of content toenable location-based contextual information. Section II describesseveral examples of different embodiments of the invention that providelocation-based and content-based information to a mobile device.Finally, section III provides a description of a computer system withwhich some embodiments of the invention are implemented.

I. Use of Geotagging of User Content to Enable Location-Based ContextualInformation

A. Wireless Positioning Methods

FIG. 1 conceptually illustrates a geographic area in some embodimentsthat has multiple wireless networks and a wireless mobile device 105that has one or more wireless transceivers that use such networks tocommunicate information and find the mobile device's location. Forsimplicity, what is referred in this specification as a mobile device, amobile station, or mobile includes user equipments such as cellulartelephones, personal communications service (PCS) telephones,wireless-enabled personal digital assistants (PDAs), wireless modems,laptop computers, tablet computers, smart phones, handheld devices thatinclude short range radios (such as IEEE 802.11 or Bluetooth® but do nothave cellular phones), digital still/video camera, or other wirelessdevices that are network capable and can generate content.

The networks include GPS, cellular (e.g., 2G, 3G, 4G), WLAN 802.11,Bluetooth®, Radio-Frequency Identification (RFID), WorldwideInteroperability for Microwave Access (WiMax), HD Radio™, Ultra-wideband(UWB), ZigBee, and 60 GHz. Each type of network can also have differentimplementations. For instance, the WLAN 802.11 standard can include802.11b, 802.11a, 802.11g, and 802.11n. The cellular network can includeCode Division Multiple Access (CDMA), CDMA2000, Wideband CDMA (WCDMA),Time Division Multiple Access (TDMA), Frequency Division Multiple Access(FDMA), and other multiple access techniques. A mobile device cancommunicate wirelessly with one or more of these networks. Thecommunication can be downlink from the network base-stations (meaningsatellites 110, WLAN or Bluetooth® base-stations 115, cellular towers120, etc.) to the mobile device, or the uplink direction from the mobiledevice to the base-stations. FIG. 1 shows the downlink direction. Themobile device receives wireless signals and processes the signals into alocation estimate. Typical information used for positioning includesGlobal positioning System (GPS) signals, Received Signal StrengthIndicator (RSSI), Angle of Arrival (AOA), Time of Arrival (TOA), TimeDifference of Arrival (TDOA), and Doppler shift. For instance,triangulation can be employed where multiple range or angle measurementsfrom known positions are used to calculate the position of a device. Wehave some prior inventions in wireless positioning methods.

A wireless positioning method that works in both outdoor and indoorenvironments and can handle multipath propagation and indirect line ofsight is described in U.S. patent application Ser. No. 12/852,443,entitled “Method and System for Determining the Position of a MobileDevice,” filed concurrently with this application on Aug. 6, 2010. U.S.patent application Ser. No. 12/833,938, entitled “Method and System forDetermining the Position of a Mobile Station,” filed Jul. 9, 2010describes a system that employs a known Radio Frequency (RF)transmission pattern to locate a receiver object. As the signal travelsbetween a transmitter and a receiver the system calculates the timedelay between the two. By repeating this process with different knowncoordinate transmitters it is possible to process the time delayinformation and calculate the unknown position coordinates of thereceiver. U.S. patent application Ser. No. 12/059,558, entitled “Methodsand Systems for Determining the Location of an Electronic Device,” filedMar. 31, 2008, published as U.S. Patent Publication 2009-0243932describes a system that employs multi-tone OFDM signals for locating anobject. As an OFDM signal travels between a transmitter and a receiverthe system calculates the residual phase difference between pilot tonefrequency pairs. By repeating this process with different knowncoordinate OFDM transmitters (or receivers) it is possible to processthe residual phase differences to calculate the unknown positioncoordinates of an OFDM transmitter (or receiver).

In U.S. patent application Ser. No. 11/940,219 entitled, “Systems andMethods of Assisted GPS,” filed Nov. 14, 2007, and published as U.S.Patent Publication 2009-0121927, describes wireless access points thatsend differential assistance data to GPS receivers which are integratedinto cellular chipsets and other chipsets. Errors caused by multipathtravel of the GPS signals are reduced by using fixed location referenceaccess point receivers. Contents of U.S. application Ser. No.12/833,938, U.S. patent application Ser. No. 12/852,443, U.S. PatentPublication 2009-0243932, and U.S. Patent Publication 2009-0121927 arehereby incorporated by reference.

B. Geotagging User Content

FIG. 2 conceptually illustrates a wireless mobile device 205 of someembodiments that creates multimedia content, tags the content withposition/orientation/timestamp information, and uploads the content to aset of location-based application servers 210 which in turn updatecontent databases 215 and user databases 220. The corresponding flowdiagram is described by reference to FIG. 3. FIG. 3 conceptuallyillustrates a process 300 for tagging and storing content generated by amobile device in some embodiments. As shown in FIG. 3, the wirelessmobile device 205 generates (at 305) content. For instance, the usertakes a picture, sends an email message, sends an SMS, records video, orrecords audio.

Next, depending on the user's security settings, the multimedia contentis geotagged (at 310). User security settings are described by referenceto FIG. 4 below. In some embodiments, geotagging is done by addinggeographical identification to media content such as photographs, video,audio, and text. In some embodiments, geotags are added to the contentas metadata. Different embodiments use one or more of latitude andlongitudes coordinates, orientation, altitude, bearing, distance, placenames, and the like to geotag content. For simplicity, throughout thisSpecification, the terms location and position are used to also refer toone or more of the latitude and longitudes coordinates, orientation,altitude, bearing, distance, place names, and the like that are used togeotag content. In some embodiments the content is timestamped (at 310),e.g., by using date and/or time, in addition or in lieu of geotagging.

In some embodiments, the user of the mobile device uses a web browser225 or program (that runs on the mobile's operating system) and anetwork 230 (cellular or other wireless networks that connect to wirednetworks) to authenticate herself to a set of networked servers 210, andupload (at 315) the multimedia content to databases 215-220 that areconnected to the networked servers. There can be several networkservers, databases and mobile devices; FIG. 2 shows only one of each forsimplicity. The servers perform different functions and can be separateprocesses that run on separate computers, or a single computer.Likewise, there are several databases that store different kind ofinformation. Each database can be stored on a single physical storagedevice or spread across multiple storage devices. In some embodiments,the networked servers or some other servers run location-basedapplications that allow the user to store (at 320) and share multimediacontent with others. The user does not have to manually label thelocation, orientation and creation time of the content if the content isgeotagged. An authorized client device or an authorized server can thenuse an application or a web browser running on its operating system toconnect to the network servers and access information, including some ofthe uploaded multimedia content. The network servers also include alocation-based contextual ad server that provides ad (i.e.,advertisement) based on the location and context of the client. Otherdatabases such as user location history database, user profile database,user usage database are also updated (at 325). Different databases usedin some embodiments are further described by reference to FIG. 5 below.

The mobile device in FIG. 2 includes one or more wireless radio devices235 with antenna 240 that transmit and receive electromagnetic waves,and baseband transceivers 245 that process the information. A processor250 controls the radios. The mobile device has a location moduleapplication 252 that processes the radio signals and calculates theposition of the mobile at any given time. The mobile device also has amagnetometer sensor 255 that provides the orientation of the mobile inthree dimensions. The memory 260 has read-only memory (ROM) for programstorage and random access memory (RAM) for running programs. Theoperating system 265 interfaces the hardware with the user and managesresources. Input devices of the mobile include keyboard and/or touchscreen 270, microphone 275, and a camera photo-sensor 280 for stillpictures and video. Output devices include a display 285, speaker 290,and wired or wireless headsets 292. The mobile device has a number ofdedicated applications 295 such as email, chat, SMS, and camera. Themobile device also has a browser 225 that is used to browse informationon the intranet/Internet and download other applications from thenetwork. There is a bus (not shown) that links all these components withthe processor and memory. A section 262 of the memory (shown separately)is dedicated to local storage of content such as pictures, video, audio,and email.

Some of the applications in FIG. 2 use location module 252 to tag thegenerated content with location information, and magnetometer 255 to tagthe content with three-dimensional (3D) orientation information. Theyalso use date and time module 248 to tag the content with a date andtime stamp. The location module in FIG. 2 provides latitude andlongitude information or some other coordinate system. In someembodiments, the location module also converts the latitude/longitudeinformation to street addresses for maps such as Google™ Maps. Thelocation coordinates, 3D orientation, and data/time information can beincluded in the generated content, coded in the filename of the content,or stored and sent in a separate metafile or message that is associatedwith the content. Thus, content generated by applications such as email,chat, SMS, social networking tools, and the camera can be complementedwith location, orientation and date/time information if the user choosesto do so by setting the appropriate security preferences. Some of thesesecurity preferences may be stored locally on the mobile, while othersmay be stored on user profile databases that are accessible to networkedservers that provide applications (e.g. email, chat, social networking).

One of ordinary skill in the art will recognize that process 300 is aconceptual representation of the operations that are performed in someembodiments. The specific operations of process 300 may not be performedin the exact order shown and described. The specific operations may notbe performed in one continuous series of operations, and differentspecific operations may be performed in different embodiments.Furthermore, the process could be implemented using severalsub-processes, or as part of a larger macro process.

C. Setting Security Preferences

In some embodiments, the mobile user can change the settings in hersecurity preferences 228 in order to enable or disable the broadcastingof her mobile's position and orientation for live tracking. FIG. 4conceptually illustrates an example of a graphical user interface 400for setting security preferences in some embodiments. As shown, the userhas disabled (by checking box 405 of general setting 415) the continuoustransmission of her location and orientation for privacy reasons. Inother words, the user has disabled live tracking of the mobile device'slocation. However, the user has set the security preference settings toenable the content that she produces (pictures, audio, video and textare shown as example) to be time-stamped (by checking the timestampboxes of content-based settings 410) and geotagged (by checking thegeotag boxes of content-based settings 410) with location/orientationinformation. Another way to achieve control of geotagging andtime-stamping of content is to do it at the application-level. This isalso shown in FIG. 4, where the user has enabled geotagging (by checkingthe geotag boxes of application-based settings 420) and time-stamping(by checking the timestamp boxes of application-based settings 420) ofemail, chat, SMS, blogging content, and the like. This level of controlallows the user to make finer trade-offs of security for benefitsassociated with location-based content and applications.

Instead of allowing the mobile device to broadcast the position andorientation continuously, the user can now control it at the level ofcontent creation. There are several different combinations of thesesettings. For instance, some embodiments allow live location tracking aswell as geotagging all content, or allow live location tracking butdisable content geotagging, etc. Some embodiments provide finer securitypreferences in the mobile device's security preferences settings, suchas being able to disable content tagging at certain times in theday/week/year or certain location areas. The geotagged contents providethe user's locations at the time-stamped times, and once the user sendsthem to a server the server can extract that information.

The user can further control privacy on the server by setting usersecurity preferences in the user profile database. These settings allowthe user to show her location to certain people that she chooses buthide it from others. Privacy can be controlled further by providing usersecurity preference settings for showing only approximate locationinstead of exact location (e.g. show city but not street address, orshow street name but not number).

D. Using Geotag and Timestamp of User Content to Determine User Locationand Provide Location-Based Information

FIG. 5 conceptually illustrates a system of some embodiments that usesgeotag and timestamp information from user-provided content to determineuser location and provide contextual location-based information.Although many examples are given below that refer to location-basedinformation as location-based advertisement, location-based informationin some embodiments is provided in response to the user requests fordirections, location of a particular service (e.g., emergency roomservice) in an area, etc. The flow diagram for content display withcontextual location-based advertising is described by reference to FIG.6.

FIG. 6 conceptually illustrates a flow diagram for providinglocation-based contextual advertising in some embodiments. As shown inFIG. 6, user of mobile A 505, user of mobile B 510, or some otherwireless or wired client use a public network 515 and provide (at 605)some identifying information or credential, such as User ID andpassword, in order to authenticate the mobile device to communicate withan application server 520 for location-based applications such as email,chat, SMS, and social networking. The application server can access usercontent database 570 and present to authorized users some of the userprovided content, as well as content 577 served by other networkedinformation servers 575 (e.g. other Internet sites such as ratings,news, weather, etc). After a client has successfully logged into theapplication server the server can access other databases that containinformation about the user (such as a user profile database 525 and auser location database 530) and provide that information (at 610) toother servers such as the location server 535 that determines theclient's location.

User profile database 525 stores user registration information for aparticular location-based application service (Email, SMS, socialnetworking, etc). Typical user information that is stored includes name,password, address, billing information, user interests and preferences.User location database (or user-location history database) 530 includesentries for position/orientation and the recording time of thelocation/orientation for each user. The location-based applicationserver takes advantage of the user's interest/preferences from the userprofile database 525 and the content that it is serving the client todetermine context for serving targeted ads. User interest andpreferences are expressed as keywords in the user profile database.Location-based application server 520 also indexes the content that isbeing seen by the client to determine context. There are many differentindexing and grouping algorithms that can be used to determiningcontext. For example, if an email says “How about Chinese food forlunch” then the words “Chinese”, “food” and “lunch” could be used todetermine the context and search for ads that contain those keywords,and further narrow the ads to those businesses that are geographicallyclose to the location of the client. If the user profile interestindicates that the user's diet is vegetarian then the application servercan narrow down the ad search further by instructing the ad server tosearch for Chinese restaurants that offer vegetarian dishes.

The location information of the mobile user is determined by a locationserver 535. The location server uses (at 620) different techniques tolocate a client. When the location is approximate the location server insome embodiments outlines an area, such as a circle with a center and aradius, or some other shape such as a rectangle or a polygon. The directapproach is where the user's security preferences allow live positiontracking and the mobile device's magnetometer/location module transmitsthe mobile device's orientation/position information to the locationserver at regular time intervals or as polled or requested by thelocation server. In some embodiments, the location server also storesthis information in the user location database 530. The live trackingoption, however, can be disabled by the settings of the securitypreferences of the mobile's user. An indirect approach is where the userdoes not allow live position tracking but does allowposition/orientation and/or time-stamping of the content that shecreates with different applications. With this approach, some or all ofthe user-provided content contain geotag and time-stamp information. Thelocation server then either extracts the position/orientation andtime-stamp information from the user-provided content as new content isadded and stores that in the user location database, or does it on thefly as needed.

In the absence of live location tracking or recent user providedgeotagged content, the location server uses other indirect means tolocate the client. In some embodiments, the IP address of the client'srequest is looked up in an IP-Address-Position database (not shown) tomap it to a location, town or zip code. Some embodiments use the addressthat is stored in the user profile database, or the lastaddress/orientation in the user location database. Some embodiments alsouse an indirect method of position calculation by doing interpolationbased on previous user location history positions. The tagged contentsprovide the position of the user at various time instances. It is thenpossible to calculate velocity and direction and use simple linearmethods or more complex algorithms (e.g. Baysian or Kalman Filtering) toapproximate the future position of the user based on previous positions.A simple example of using only the last two position measurements is asfollows. Suppose the user was at (x, y) position (0, 0) two hours agoand at position (1, 0) one hour ago. Then the user is traveling 1 unitin the x direction each hour and current position can be estimated to be(2, 0).

The output of the location server can be a circular area with a centerand a radius, where the radius defines the confidence in the position.The user (or users if the ad is targeted to a group of people) can beanywhere in the circle. The greater the confidence in the position, thesmaller the radius is. When the mobile device is allowing locationtracking, the mobile device orientation and position are very currentand we can even infer which store the mobile device is facing towards.When position tracking is disabled but the user has uploaded a geotaggedcontent, the accuracy of the user's location decreases (and the radiusincreases) with increasing elapsed time since content was created andtime-stamped. Therefore, the time of the mobile's position/orientationmeasurement or tagging demonstrates the degree of confidence in theposition/orientation.

FIG. 5 also shows a location-based ad server 540 that provides ads tousers of the location-based application. The location-based applicationserver 520 uses content indexing and user profiles to provide (at 615)the ad server with context. Location server 535 provides locationinformation to location-based ad server 540. The ad server can access anad content database 545 which stores information such as business names,addresses, and ads and coupons for their products and services. The adcontent database is routinely updated by an administrator with new andup-to-date ad content. This database may be a relational database sothat the ads can be searched for keywords and the location of thebusiness. The ad server searches the ad content database to find (at630) contextual advertisements that match the user context and location.Advertisers that pay a premium get featured ads with a betterpresentation such as higher listing order (if several ads arepresented), better screen position, larger fonts, etc.

The location-based ad server then sends (at 630) to the location-basedapplication server ad information such as the text of the ads, thepositions of the businesses/services, their telephone numbers, links totheir ratings by users, links to the advertisers' web sites, and adcoupons that can be saved for later redemption.

The application server also has the content, the metadata of the content(which includes position/orientation and timestamp information), users'profile information and the location of the users. In some embodiments,the application server optionally adds the user's name to the text ofthe ad to personalize it and uses (at 635) different presentationmethods for display the content together with the advertisement. Someexamples are described below by reference to FIGS. 11-15. Theadvertisement is displayed as a pop-up, a banner ad, or an embedded partof the display window. When there are several relevant businesses in thevicinity of the user some embodiments display the closest one whileother embodiments rank and display the ads in the order of closeness tothe user's location. In some embodiments, the application server uses amap server 550, which can access a geographical map database 555, togenerate static or dynamic maps, where the position/orientation of thecontent/users and the ad location are displayed in a single map. Someembodiments show the radius that defines the confidence in the positionas a circle around the location. The map server in some embodimentsgenerates a map with driving directions (or uniform resource locatorlinks to driving directions) that starts with the location of the userand ends at the location associated with the advertisement. The ads alsoinclude the telephone number of the advertiser in some embodiments sothat the mobile user can call them. This can be automated if the adincludes a Uniform Resource Locator (URL) or a similar link thatincludes the advertiser's telephone number as a parameter and points toa Voice over Internet Protocol (VoIP) server 590 so that when the linkis clicked it will invoke an application (e.g. Skype) and make the call.

A URL link shortener 560 is used by the application server in someembodiments to create unique short URL aliases that redirect to longURLs that have name/value pairs (e.g. for GPS coordinates, orientation,date, and time). These shorter links are more suitable for embedding inads and maps. The application server also formats the ad and provides aURL link that uses a rating server on the Internet that provides ratingson the recommended ad products/businesses/service. The users can alsouse this link to provide their ratings if they use theproduct/business/service.

In some situations more than one user are collaborating on a task (suchas an email thread between groups of people discussing where to go forlunch). There are at least three options for targeting the location ofthe ad: 1—Target the sender's locations; 2—Target the recipient'slocations; and 3—Target the location of all or a subset of the group.Which option is pursued depends on the application and the context. Thedisclosed system can handle all of these cases. FIG. 6 shows option 3where the application server and the location server calculate thelocation of all the users involved in the task and find (at 625) theaggregate central location to all of them, and use that as the locationfor the group. The ad is then based on the location of all the usersinvolved in the task, but still personalized to each user (i.e.direction links are personalized based on the position of each user,each user sees their name on the ad targeted to them, etc.). Theapplication server also uses the map server 550 to generate a mapshowing all the collaborating users/content. In another embodiment, thecontext of the collaboration task determines the location for thelocation-based ad. For instance, if there is discussion between twopeople in Santa Monica about going on a vacation to Cancun, Mexico, thenthe application server may override the location of the two users andtarget the ads based on the location from the context (i.e. use Cancunas the location and provide ads about things to do in Cancun, as opposedto ads for travel agents in Santa Monica that are close to them).

In FIG. 5 a user usage database 565 records the interactions of userswith the location-based application and the location-based ad server.For example, the database logs requests with date/time, uploadedcontent, searches, clicked ads/coupons, and transactions. In someembodiments, data mining server 578 accesses the user usage database 565and the user's location history (e.g., dates/times and locations fromthe user location database) to analyze how effectively the userinteracts with the application and how successful the targetedadvertisements are. The mining server may also employ image/videocontent retrieval and classification methods to classify the content ofthe user's image/video content, as well as indexing the textualcontribution of the user to infer about the interests of the user. Themining server can then update the user profile database in order toupdate the user's interests and preferences. The ad server andapplication server can then use the profile database to personalize andtarget the ads for the user.

FIG. 5 demonstrates the example for one location-based applicationserver (e.g. social networking). In practice users register withdifferent services for different applications (email, social networking,etc) and so user databases and content are distributed across multipleservers and databases. Even for the same application a logical databasecan be implemented with multiple databases across several physicalservers.

FIG. 7 conceptually illustrates a process 700 performed by a mobiledevice to send content to one or more servers and receive location-basedinformation in some embodiments. In some embodiments, process 700 isutilized by a mobile device as described in FIGS. 2 and 5 above. In someembodiments, process 700 is performed to tag content while the usersecurity settings do not allow regular transmission of location (e.g.,when box 405 in FIG. 4 is checked). In other embodiments, process 700 isperformed to tag content while the user security settings allow regulartransmission of location.

As shown, the mobile device creates (at 700) multimedia content whene.g., a user takes a picture, sends SMS or email, records audio, etc.Next, the mobile device performs (at 710) authentication with anapplication, database, or web server to upload content.

The process then determines (at 715) whether the user security settingsauthorize content sent to the server to include geotags such aslocation, orientation, etc. and/or timestamp information. When thesecurity settings do not authorize such tagging, process 700 sends (at720) content to the server without the corresponding tags. The processthen proceeds to 730 which is described below.

On the other hand, when the security settings allow tagging of some orall of content data, the process provides (at 720) the correspondinginformation (e.g., position, orientation, and/or timestamp) for thecontent. At described by reference to FIG. 4 above, security settings insome embodiments allow a user to selectively determine which type ofcontent or content generated by which application can be geotaggedand/or time tagged.

The process then sends (at 725) content and the correspondinginformation (e.g., in the form of metadata) to the server. At a latertime, the process receives (at 730) location-based information (e.g.advertisement) from one or more authorized providers associated with theserver. The process then ends. As shown, even when the mobile devicesends (at 720) content that is not tagged, if the security settings werepreviously set to allow tagging, the security settings were set to allowtagging for different type of content, or live tracking is on, themobile device can still receive location-based information (at 730).

One of ordinary skill in the art will recognize that process 700 is aconceptual representation of the operations that are performed in someembodiments. The specific operations of process 700 may not be performedin the exact order shown and described. The specific operations may notbe performed in one continuous series of operations, and differentspecific operations may be performed in different embodiments.Furthermore, the process could be implemented using severalsub-processes, or as part of a larger macro process.

FIG. 8 conceptually illustrates a process 800 performed by one or moreservers (e.g., by the location-based application server 520) in someembodiments to receive content from a mobile device and provide theinformation to one or more authorized servers. As shown, the processreceives credentials from the mobile device and authenticates (at 805)the mobile device to access the servers. Next, the process receives (at810) multimedia content from the mobile device. As described above,depending on the security settings of the mobile device some or all ofthe content is geotagged and/or time tagged.

Next, the process stores (at 815) content and the correspondinginformation in one or more databases (e.g., user location history 530,user profile 525, user usage 565 databases). In some embodiments, thecorresponding information is stored in the form of metadata associatedwith the multimedia content. In some embodiments, the process associatesadditional security information (e.g., as metadata) to indicate whichother users or servers are authorized to access content that is receivedfrom this particular mobile device or this particular user. In someembodiments, this additional security information is received from themobile device. In other embodiments, the additional security informationis retrieved from a database (e.g., the user profile database 525).

The process then receives (at 820) a request to access the contentand/or the corresponding information. The requesting party can be thesame mobile device requesting to download the information at a latertime or an ad server (e.g., location-based ad server 540) requesting thecontent and its associated metadata to create location-based informationfor the same mobile device or a different client. The process determines(at 825) whether the security setting associated with the content and/orthe corresponding information allow providing the requested informationto the requesting party. When the security setting associated with thecontent does not allow the requesting party to receive the information,the process ends. Otherwise, when the security settings associated withthe content allow the requesting client to receive part or all of therequested information, the process provides (at 830) the authorizedcontent and/or the corresponding information to the requesting party.The process then ends.

One of ordinary skill in the art will recognize that process 800 is aconceptual representation of the operations that are performed in someembodiments. The specific operations of process 800 may not be performedin the exact order shown and described. The specific operations may notbe performed in one continuous series of operations, and differentspecific operations may be performed in different embodiments.Furthermore, the process could be implemented using severalsub-processes, or as part of a larger macro process.

FIG. 9 conceptually illustrates a process 900 performed by one or moreservers in some embodiments to receive content previously generated by amobile device and provide location-based information to the same mobiledevice or different clients. In some embodiments, process 900 isperformed by a server (e.g., an ad server 540 or a map server 550) thatreceives the content and/or metadata related to location or time andprovides location-based information (e.g., ads or directions) to aclient. In some embodiments, process 900 is started when a server isgoing to send location-based information to a target mobile device orwhen the server has received request for information from a requestingmobile device.

As shown, the process authenticates (at 905) the requesting server withthe content server. The process then receives multimedia content and/orthe associated metadata from the content server. Next, the processextracts (at 915) location and/or time information from the receivedcontent. The process then customizes (at 920) the location-basedinformation for the target or the requesting mobile device based on theinformation stored in the profile of the user of the mobile device.

The process then sends (at 925) the customized information (e.g.location-based advertisement, maps, directions, etc.) to the target (orthe requesting) client. The target or the requesting client is eitherthe same mobile device that generated the content and the associatedmetadata or a different client. Several examples of different scenariosfor sending location-based information to one or more mobile devices aredescribed in the following sections. The process optionally provides (at930) a link to a VoIP server 590 for the receiving mobile device to makea call to a service provider. The process then ends.

One of ordinary skill in the art will recognize that process 900 is aconceptual representation of the operations that are performed in someembodiments. The specific operations of process 900 may not be performedin the exact order shown and described. The specific operations may notbe performed in one continuous series of operations, and differentspecific operations may be performed in different embodiments.Furthermore, the process could be implemented using severalsub-processes, or as part of a larger macro process.

FIG. 10 conceptually illustrates a process 1000 performed by one or moreservers to provide location information of a mobile device to arequesting party in some embodiments. Process 1000 is performed, e.g.,by a location server 535 that receives a request from a location-basedad server 540 or a map server 550 that is performing process 900. Asshown, the process receives (at 1005) a request for location and/orother similar information (e.g., orientation, closes address, latitude,longitude, etc.). The process determines (at 1010) whether therequesting party is authorized to use the requested information aboutthe mobile device.

When the process determines that the requesting party is not authorizedto receive the requested information, the process ends. Otherwise, theprocess determines (at 1015) whether the security settings of the mobiledevice allow live tracking and live tracking data is available. When thesecurity settings allow live tracking, the process uses (at 1020)information received from the mobile device to determine location orother related information of the mobile device. Depending on the mobiledevice's settings, when the live tracking is allowed, the locationinformation is broadcasted by the mobile device, periodically sent to acertain servers, or is provided upon request. The process then ends.

Otherwise, when live tracking is not allowed, the process determines (at1025) whether security settings of the mobile device allow content toinclude some or all of the requested information (e.g., whethergeotagging of some content is allowed). When content is allowed toinclude the requested information, the process uses (at 1030) theinformation associated with the content to determine the location and/orother requested information of the mobile device and sends theinformation to the requesting party. The process then ends. In someembodiments, the process considers the time the content was generatedand refines the location estimation based on how old the content is. Insome embodiments, the process uses prior locations of the mobile deviceand determines an estimate of the direction and speed of the movement ofthe mobile device. The process then uses that information to interpolateor extrapolate the location of the mobile device. In some embodiments,the process identifies the location of the mobile device by a circle andas the content becomes older, the radius of a circle becomes larger.

When content is not allowed to include the requested information, theprocess uses (at 1035) indirect methods such as IP address, addressstored in a profile database, last known address/orientation,interpolation/extrapolation based on previous user location history todetermine the mobile device's location or other requested informationand sends the information to the requesting party. The process thenends.

One of ordinary skill in the art will recognize that process 900 is aconceptual representation of the operations that are performed in someembodiments. The specific operations of process 900 may not be performedin the exact order shown and described. The specific operations may notbe performed in one continuous series of operations, and differentspecific operations may be performed in different embodiments.Furthermore, the process could be implemented using severalsub-processes, or as part of a larger macro process.

II. Examples of the Use of Geotagging to Determine User Location andProvide Location-Based Advertising

Several embodiments that use geotagging and timestamping to determineuser location and provide location-based advertising are described inthe following sub-sections.

A. Social Networking Site

FIG. 11 conceptually illustrates an example interface of someembodiments for a user called “Sue Welsh” that uses a mobile device inFIG. 5 to connect to the networked server of FIG. 5 and access some ofthe information in the databases in FIG. 5. In this example thelocation-based application on the server 520 in FIG. 5 provides acommunication web site for social networking (e.g. Facebook®, MySpace™,YouTube®, Twitter™ and LinkedIn®). The user sees really simplesyndication (RSS) news, pictures, audios, videos, SMS, and comments fromher chosen circle of friends. Three such friends are shown in FIG. 11;“John Doe”, “Bill Smith” and “Jill Wright”. Sue sees the latest 4pictures (1120) that were uploaded by John Doe. The date and time wheneach picture is taken (and time-stamped) is shown next to the picture.These are different from the date and time when the pictures areuploaded to the web site. This is one of the benefits of including thecreation date and time information with the content.

The tagged location of each picture (P1, P2, P3 and P4) is alsodisplayed on a map 1105, with the arrows showing the compass direction(a 3D map can also display the tilt). In some embodiments, the map alsoshows the creation time of the content. Sue and her friends see an SMSmessage from Sue. The message also has a small URL 1140 at the end ofthe message that indicates her location/orientation when she sent themessage. That location and compass orientation is also shown on the map(S1). In some embodiments, the location URL uses URL parameters todistinguish location parameters, angle and tilt. A URL parameter beginswith a question mark (?) and takes the form of a name/value pair that isappended to the URL. If more than one URL parameter exists, eachparameter is separated by an ampersand (&). In some embodiments theapplication server uses a special reserved identifier (The string “Loc:”in this example) to differentiate the user's location URL from otherURLs that may be embedded in the message. Bill Smith has also added avideo (1110) with a comment. The creation time of the video is shownbelow it. Again this is different from the upload date and time that isalso shown. Also shown on the map are the tagged location andorientation of the mobile whilst taking the video (V1). Jill Wright hasalso added a comment 1115. Her location and orientation when she enteredthe comment is also shown on the map (C1). The user interface may alsodisplay the location range of the user with a circle, where the radiusis proportional to the difference between current time and the content'stimestamp.

As shown in FIG. 6, the location server uses a number of methods todetermine the location of the user. These include position tracking,metadata of user's tagged content, user location history, user profileand user's IP address. The application server in FIG. 5 also has anassociated ad server that provides location-based contextualadvertisements to the client that is viewing a page. In FIG. 11 bothBill and John have set their mobile's security preference settings suchthat their mobile's position is regularly transmitted to the locationserver of the social networking service. Jill and Sue, however, havedisabled that feature but enabled content tagging. Some of the securitypreferences are specific to certain applications/web services and arestored in the user profile database of the application server. Forexample, Bill and John can specify which friends should be able to seetheir locations on the social networking site, and those settings arestored in the user profile database in FIG. 5. In this example they haveallowed Sue to see their locations. Hence, their most up-to-datelocations, as well as their latest content locations, are marked on themap that Sue sees. There could also be additional controls that allowsthem to hide their locations at specified times and dates.

The following is a location-based contextual ad example. Suppose thatwhen Sue Welsh logs in with her mobile to the social networking site herlocation is, “1500 Ocean Ave, 90401-2119, Santa Monica, Calif., USA”.After logging in she posts the SMS message 1125 “I am going to get somecoffee” to her friends on the social networking site. Her mobile'ssecurity preferences allow geotagging the position/orientation of hercontent and her user profile on the social networking site allows allher friends to see the tagged information. The application serverindexes Sue's message and sends the keywords “Coffee” and “Shop” to thead server. The location server also obtains Sue's Santa Monica locationfrom the geotagged SMS message (extracts longitude and latitudeinformation and uses the map database to convert them to streetcoordinates). The accuracy of this tagged location is high since thetimestamp on the SMS message is close to current time. The ad serverthen searches the ad database for “Coffee”, “Shop” and “Santa Monica”,finds several results and uses the map database to calculate the closestcoffee shop to Sue's location. The ad server then finds the ad for theclosest coffee shop: “You are invited to free pastries at Famous CoffeeShop at 900 Ocean Ave, Santa Monica, Calif.”. This ad and the locationassociated with the shop are sent to the application server.

The application server then personalizes the ad message by adding Sue'sname: “Dear Sue, You are invited to free pastries at Famous Coffee Shopat 900 Ocean Ave, Santa Monica, Calif.”. The application server sendsthe location of the coffee shop to the map server. The location of therecommended coffee shop is then added to the map (A1). Similarly, the adserver would serve different location-based ads to Sue's friends (“JohnDoe”, “Bill Smith” and “Jill Wright”) based on each of their locations,their contexts, and user profile database entries. This ad exampleillustrates the placement of an ad directly on the social networkingsite of a user based on the user's location and the context of hermessage. Other ads could be recommended based on the data miningserver's analyzing of the demographic, behavior and location of theuser's circle of friends. The ad server can also send additionalinformation to the application server for formatting and display suchthat the advertiser's ad has links for directions, telephone calling,user ratings, advertiser's web site and advertiser's coupons (e.g. seeFIG. 15 below).

B. Email

In some embodiments, the application server in FIG. 5 is an email serverthat supports an advanced email protocol (such as Internet Mail AccessProtocol (IMAP)) where the email stays on the server. Then the contentdatabase 570 in FIG. 5 includes text emails as well as emailattachments. In some embodiments, the mobile devices use a browser or anemail client to communicate with the email application server. In oneconfiguration users who are sending each other email are using the sameemail service (e.g. Yahoo!® Mail) and so they are using the sameapplication server and their email contents are stored in the samelogical databases. In another configuration, the users are usingdifferent services (e.g. Yahoo!® Mail and Gmail™). In this configurationwhen a Yahoo® email account user sends an email to a Gmail™ account bothYahoo® and Gmail™ servers keep a copy of the email in their databases.So while the application/ad servers for the two users are differentsystems, their shared emails with context and geotagging information areaccessible to both systems. Also note that a single email that has oneattachment can include two sets of position/orientation/timestampinformation; one for where/when the email is sent and one for where/whenthe attachment was created.

FIGS. 12 and 13 conceptually illustrate examples of location-basedadvertising for email in some embodiments, where the sender'sposition/orientation is shown in the sender's email signature and the adtargeting the recipient is shown at the bottom of the email. In thiskind of collaborative scenario there are at least three options forchoosing the address to target the ads to: 1—Target the email sender'slocations; 2—Target the email recipients' locations; and 3—Target all ora subset of all the locations (sender's and the recipients′). In thisexample the email application server determines that this iscollaboration so it targets the ads to both locations if they are known.As shown “Dave” receives (at 1205) an email from “John Yu” about a lunchmeeting, but the email is not geotagged and does not provide locationinformation about the sender. The location server uses one of the othermethods (other than using geotags) to find the recipient's location(user location history, user profile, IP address). However, thesemethods are typically less accurate and are at the scale of the zipcode. The email application server indexes the email, determines contextwith keywords such as “lunch” that appear inside the email, and uses anad server to make a restaurant recommendation to Dave. There could alsobe a list of restaurant ads for Dave at the bottom of the email. Sincethere is no location information about the sender the received email hasa default signature: “Sent via my mobile phone”.

Such signatures are often static and are stored in a file. One optionfor geotagging email is for either the client or the email applicationserver to embed the GPS Coordinates, the compass orientation and tiltdegree, and the client's timestamp in the email signature: “Sent via myMobile Phone” Loc: GPS Latitude, Longitude Coordinates: 34.012693,−118.496485, pointing North, tilt 20 o, on Jul. 4, 2009, 10:56:35 am.”

The examples of FIGS. 12 and 13 use the “GPS decimal notation withnegative numbers for South and West”. There are different notations thatcould be used that are equivalent such as “Degrees and decimal minuteswith N, S, E or W suffix for North, South, East, West” or “Degrees,minutes and decimal seconds with N, S, E or W suffix for North, South,East, West”.

In these examples, the location module in FIG. 2 provides the GPScoordinates to the email client program, the magnetometer provides theorientation, and the date/time module provides the date and time. Theemail client program then adds the position, orientation, and date/timestamps to the signature file. If a browser is used then a softwarecomponent such as a plug-in or Active X component needs to be installedthat can perform similar tasks for the signature file. Other embodimentsdo email geotagging by sending the position/orientation/time-stampinformation as an attachment with a special reserved name identifierthat the email server knows about and recognizes, and then the emailserver modifies the signature or adds this information somewhere in theemail body. If html email and dynamic images are supported then theemails can be presented in html with dynamic images for locations andmaps. If, however, this is not the case then the emails can include URLlinks that point to dynamic maps and/or include static images of themaps as attachments.

FIG. 12 also shows examples when the sender, “John”, adds hisposition/orientation/time-stamp information to the email signature andit is displayed using different presentation methods. For instance, FIG.12 shows (at 1210) an example of textual representation of thisinformation. FIG. 12 also shows (at 1215) an example of a short aliasURL for a longer URL that points to a server, where the URL hasname/value pairs that the server extracts for position, orientation andtime/date. The user then clicks on the short URL to see a graphicalpresentation of position/orientation and date/time. FIG. 12 also shows(at 1220) an example of an embedded map display that is similar to whatwould be displayed if the URL in the signature 1225 is clicked. Thistranslation and formatting of the user's location/orientation andtimestamp are done by the email system's location server, the emailapplication server, and the corresponding map server 550, as shown inFIG. 5. The examples shown at 1210-1220 show a different ad compared tothe example shown at 1205 since in the examples 1210-1220 the sender hasprovided his location through the email signature and the ad can betargeted to the location of both the sender and the receiver. In theexample shown at 1220 the location of the receiver (“Dave”) is not shownon the map since he has not yet provided a tagged email response and theemail system only knows a coarse position for him such as a zip code.

FIG. 13 conceptually illustrates the example 1305 when the recipient“Dave” replies to the sender “John” and there are two signatures withlocation information. In the Examples of FIG. 12 the ad text ispersonalized for the recipient “Dave”. In FIG. 13 the ad text ispersonalized to “John”, since he is the recipient of the reply. In FIG.13 the email application server determines that this is collaboration soit uses the location server to extract both locations from the twosignatures with the string identifier “Loc:” It then sends bothlocations (or a single aggregate location) and information to the adserver. The ad server then recommends ads that are close to both theirlocations. The map server also generates a map 1310 with the locationsof both users and all the ads.

The location URL can use URL parameters to distinguish location geocodeparameters, angle, tilt, date and time. A URL parameter begins with aquestion mark (?) and takes the form of a name/value pair that isappended to the URL. If more than one URL parameter exists, eachparameter is separated by an ampersand (&). The email system and itslocation server can use a special identifier (The string “Loc:” in thisexample) to differentiate the user's location URL from other URLs thatmay be embedded in the message. For example the following URL hasembedded the GPS coordinates of the address in the example shown at 1210into a URL that points to a Google map server:

http://maps.google.com/maps?f=q&source=s_q&hl=en&geocode=&q=1500+Ocean+Ave,+santa+monica&sll=34.014001,-118.496475&sspn=0.008253,0.01929&ie=UTF8&ll=34.01441,-118.496475&spn=0.008253,0.01929&z=16&iwloc=A

Some embodiments parse the orientation and tilt information and use themto display the orientation on a 2D or 3D map. There are URL linkshortener services such as www.Tinyurl.com® that take the above long URLand create a shorter alias that redirects to the original long URL. Theexample shown at 1215 illustrates the result of this approach. Thefollowing is an example of the steps that some embodiments take toimplement examples of FIGS. 12 and 13.

When a user sends an email the mobile's email client embeds parametersfor GPS latitude, longitude, magnetometer direction, magnetometer tilt,date and time into the email signature with a specified format that ispreceded with a reserved identifier string such as “Loc:”.Alternatively, some embodiments send the tagging parameters as anattachment in a specified file name to the email application server andlet the server modify the signature file. The email application serverreceives the email.

Next, the email recipient logs into the email application server andrequests to see the tagged email in her inbox. The email applicationserver then downloads the requested email; its location server parsesthe signature to look for the string identifier (e.g. “Loc”), extractsthe geotagged parameters, and sends the location parameters to theapplication server. (If the tagging parameters are sent as an attachmentthen the location server downloads the attachment, extracts theparameters and sends them to the application server). (4) The emailapplication server uses a map database and a link shortener to generatea short URL for the position of the sender and modifies the sender'ssignature with this URL.

The email application server then indexes the email's text to definecontext and sends the context and the recipient's information to the adserver. The email application server also determines if the email iscollaborative. If the signatures of the other users are included in thereplies in the threaded email then the application server can extracttheir positions too. Otherwise, it asks the location server to find theposition of the other users from other means such as location history,user profile, or location tracking. Then the application server findsthe aggregate location of all users and passes that information and theusers' information to the ad server.

Next, the ad server uses the context and the location to findlocation-based contextual ads and passes the ad information to the emailapplication server. Next, the email application server formats the adsfor display at the client. For a display like the Example shown at 1215the email server can personalize the text of the ads for the user. Insome embodiments, the email server also uses a map server to generate amap and a link shortener to insert a short URL pointer for the map inthe ad. For a display like the example shown at 1220, the dynamic map isinserted in the email, provided html email is supported. The map canreplace the sender's signature as shown, or it can replace the ad'slink, or it can replace both. The example shown in FIG. 13 is similarexcept that the location of more than one user (or an aggregatelocation) is passed to the ad server and the map server.

When an email thread is send to multiple recipients, personalization ofthe text of the ad (i.e. showing the name of each recipient before thead) is more involved. Some embodiments do not personalize the text soall the recipients see the same text. Alternatively, some embodiments,for web-based email systems send a URL for the ad and that URL includesa parameter for the User ID, so that when the recipients click on thelink they would see different personalized texts with the names that areassociated with their User IDs.

FIG. 14 conceptually illustrates a process 1400 performed by one or moreservers (e.g. an email server 520) to provide location-based informationto an ad server in some embodiments. As shown, the process receives (at1405) a message (e.g., an email) that includes information such asgeotagged content, text that infers location, etc. The processdetermines (at 1410) whether the security settings of the user's mobiledevice allow providing location information to an ad server. In someembodiments, this security information is accompanied as metadata withthe content. In other embodiments, this information is retrieved from adatabase such as user profile database 525. When the security settingsdo not allow providing location information to the ad server, theprocess ends. Otherwise, the process extracts (at 1415) the locationand/or time information from the content and provides it to therequesting party (e.g., the ad server). In some embodiments, thelocation information is extracted by using a location server.

Next, the process determines (at 1420) whether content is part of acollaborative message. When the content is not part of a collaborativemessage, the process ends. Otherwise, the process determines (at 1425)whether location information can be extracted or inferred from the otherparties' messages. When such location information can be extracted orinferred for some or all of the parties to the collaborative message,the process determines (at 1430) those parties' location informationbased on the received content. The process then provides (at 1440) thelocation information of the other parties to the ad server. The processthen ends.

When location information cannot be extracted or inferred from some orall other parties, the process requests (at 1435) a location server toprovide the location information from other means (e.g., from pasthistory, user profile, location tracking, etc.) for those parties. Theprocess then proceeds to 1440 which is described above.

Some embodiments consider the time an email was sent in order todetermine the reliability of using the email for location determination.Other embodiments use the context of the email to determine a timeperiod for which the email is to be used for location determination.

One of ordinary skill in the art will recognize that process 1400 is aconceptual representation of the operations that are performed in someembodiments. The specific operations of process 1400 may not beperformed in the exact order shown and described. The specificoperations may not be performed in one continuous series of operations,and different specific operations may be performed in differentembodiments. Furthermore, the process could be implemented using severalsub-processes, or as part of a larger macro process.

C. SMS

A popular form of social networking is to send SMS text messages to alist of contacts and to receive short messages from possibly anotherlist of contacts. Twitter™ is one such example, where users can send andreceive short messages, called tweets, via the Twitter™ web site, SMS orexternal applications. It is possible to use the location of the user asspecified in her profile as the originating location of the message andshow that on a map. But this is not accurate since profiles are usuallyspecified at the city level. They are also static files and mobile usersdo not change them as they move around. SMS has limitations on thenumber of characters that can be sent. If the message extends beyond thelimit it is truncated. Thus, it is better to use the small URL aliasmethod for including position information in the SMS text message (e.g.In our Santa Monica address example one should usehttp://tinyurl.com/lph3kl instead of the original long map URL, streetaddress, or the GPS coordinates since it is the shortestrepresentation). If orientation and client-side time-stamping are alsoto be carried out they have to be shortened with aliasing or hashingmethods because of the number of characters limitation. This shorteningstep was carried out on the server-side for email, but for SMS it has tobe carried out on the client-side because of the limitations on thenumber of characters.

FIG. 15 conceptually illustrates examples of location-based SMS messagesand location-based advertising with links to directions, VoIP calling,user ratings, advertiser's web site, and advertiser's coupons in someembodiments. The example of FIG. 15 shows (at 1505) an SMS conversationbetween two mobile cell phone users, Sam Smith and Dave. When Sampresses send for his first SMS message on the screen the mobile device'slocation module first makes a separate request to a server and its linkshortener to convert the GPS coordinates to a short URL, where the URLis the equivalent map pointer. The location module then receives theshort URL and passes it onto the mobile device's client SMS programwhich in turn adds it to the SMS message. Likewise, when Dave replieshis mobile's location module adds his location's short URL to the SMSmessage. The location-based SMS application server indexes the SMScontent of the messages and recognizes the keywords “restaurant” and“Chinese”. The SMS application server sends these context keywords tothe location-based contextual ad server. The application serverdetermines that this is a collaborative task between two users. Sinceboth users have provided their locations to the SMS server the servercan send both their locations (or one aggregate location) to the adserver. The ad server can then send recommend Chinese restaurants thatare close to both their locations.

The SMS application server receives these recommendation and formatsthem for display. In this example, two restaurant are recommended (A1and A2), where the closer restaurant (A1) is shown first in the ad list.The SMS application server formats the ads and provides individualizeddriving directions to the restaurants for each of the two users since itknows both their locations. The SMS application server also providedlinks to the advertisers' web sites and coupons, as well as a “call”link that includes the advertisers' telephone number and points to aVoice over Internet Protocol (VoIP) server so that when it is clicked itwill invoke an application (e.g. Skype) and make the call. Theapplication server also provides a URL link that uses a rating server onthe Internet that provides ratings on the recommended restaurants. Ifthe two users go to one of the recommended restaurants they could alsouse this link to provide their ratings.

FIG. 15 also shows (at 1510) that the map server has generated a map1515 that shows the position of the two mobiles at the time oftransmitting the SMS messages, and the location of the ad server'srecommended restaurant. This image is transmitted to the client withMultimedia Messaging Service (MMS) since SMS does not support images.Note that some VoIP systems (e.g. Skype's peer-to-peer method) allowtext messaging between users. Thus, another application for the systemis that VoIP users can send messages to each other as in FIG. 15, andreceive location-based contextual advertising. The display of FIG. 15 isonly one example user interface for viewing SMS messages. Someembodiments display SMS messages in a format that is more similar toemail, as shown in FIG. 12. Likewise, some embodiments display theemails of FIG. 12 in a format that resembles FIG. 15.

D. Weblog

Weblogs or blogs are web commentaries. Personal blogs are ongoingcommentary by an individual. Some social networking sites allow users toshare comments. Really Simple Syndication (RSS) is also a popular webfeed format to publish web documents that are frequently updated such asblog entries, news, audio and video. Geotag information can be includedas a short URL in the web comments, or included in a metadata file,which is typically in XML format. Readers of the blogs can then find outthe location of the author when the blog entry is made. Readers can alsoplace their comments on blogs/newsgroups, where their tagged locationinformation is also included.

E. Image/Video

When the mobile device's camera takes a picture or is finished taking avideo the date and time module notes the date/time, the magnetometersensor determines the orientation, and the location module calculatesthe coordinates of the mobile device. If for any reason there are nowireless signals that can locate the mobile device, the last knownposition of the device can be used. Different embodiments store thisinformation in different ways. Some embodiments geocode the image/videoby storing the position and orientation information in special binaryfields of the image header, which is later read by image readerprograms.

Some embodiments add the orientation, longitude and latitude informationto a metadata file that is associated with the image/video such as theExchangeable Image File Format (EXIF). Some embodiments, rename theimage/video where the name of the file includes identifiers and valuesfor the orientation, latitude, longitude, date and time.

Some embodiments physically stamp the information onto the image/videoso that it is visible to the human eye at the bottom or top of theimage/video. This option is more difficult for a server to process andextract the information. Geotagged images/videos can then easily beuploaded to online content albums and organized/labeled.

III. Computer System

Many of the above-described processes and modules are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as “computerreadable medium” or “machine readable medium”). These instructions areexecuted by one or more computational elements, such as one or moreprocessing units of one or more processors or other computationalelements like Application-Specific ICs (“ASIC”) and Field ProgrammableGate Arrays (“FPGA”). The execution of these instructions causes the setof computational elements to perform the actions indicated in theinstructions. Computer is meant in its broadest sense, and can includeany electronic device with a processor (e.g., moving scanner, mobiledevice, access point, etc.). Examples of computer readable mediainclude, but are not limited to, CD-ROMs, flash drives, RAM chips, harddrives, EPROMs, etc. The computer readable media does not includecarrier waves and/or electronic signals passing wirelessly or over wiredconnection.

In this specification, the term “software” includes firmware residing inread-only memory or applications stored in magnetic storage that can beread into memory for processing by one or more processors. Also, in someembodiments, multiple software inventions can be implemented as parts ofa larger program while remaining distinct software inventions. In someembodiments, multiple software inventions can also be implemented asseparate programs. Finally, any combination of separate programs thattogether implement a software invention described herein is within thescope of the invention. In some embodiments, the software programs wheninstalled to operate on one or more computer systems define one or morespecific machine implementations that execute and perform the operationsof the software programs.

FIG. 16 conceptually illustrates a computer system 1600 with which someembodiments of the invention are implemented. For example, the mobiledevice described above by reference to FIGS. 1, 2, and 5 and the serversdescribed above by reference to FIG. 5 may be at least partiallyimplemented using sets of instructions that are run on the computersystem 1600. As another example, the processes described by reference toFIGS. 3, 6, 7, 8, 9, 10, 14 may be at least partially implemented usingsets of instructions that are run on the computer system 1600.

Such a computer system includes various types of computer readablemediums and interfaces for various other types of computer readablemediums. Computer system 1600 includes a bus 1610, at least oneprocessing unit (e.g., a processor) 1620, a system memory 1630, aread-only memory (ROM) 1640, a permanent storage device 1650, inputdevices 1670, output devices 1680, and a network connection 1690. Thecomponents of the computer system 1600 are electronic devices thatautomatically perform operations based on digital and/or analog inputsignals. The various examples of user inputs described above may be atleast partially implemented using sets of instructions that are run onthe computer system 1600 and displayed using the output devices 1680.

One of ordinary skill in the art will recognize that the computer system1600 may be embodied in other specific forms without deviating from thespirit of the invention. For instance, the computer system may beimplemented using various specific devices either alone or incombination. For example, a local Personal Computer (PC) may include theinput devices 1670 and output devices 1680, while a remote PC mayinclude the other devices 1610-1650, with the local PC connected to theremote PC through a network that the local PC accesses through itsnetwork connection 1690 (where the remote PC is also connected to thenetwork through a network connection).

The bus 1610 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of thecomputer system 1600. In some cases, the bus 1610 may include wirelessand/or optical communication pathways in addition to or in place ofwired connections. For example, the input devices 1670 and/or outputdevices 1680 may be coupled to the system 1600 using a wireless localarea network (W-LAN) connection, Bluetooth®, or some other wirelessconnection protocol or system.

The bus 1610 communicatively connects, for example, the processor 1620with the system memory 1630, the ROM 1640, and the permanent storagedevice 1650. From these various memory units, the processor 1620retrieves instructions to execute and data to process in order toexecute the processes of some embodiments. In some embodiments theprocessor includes an FPGA, an ASIC, or various other electroniccomponents for execution instructions.

The ROM 1640 stores static data and instructions that are needed by theprocessor 1620 and other modules of the computer system. The permanentstorage device 1650, on the other hand, is a read-and-write memorydevice. This device is a non-volatile memory unit that storesinstructions and data even when the computer system 1600 is off. Someembodiments of the invention use a mass-storage device (such as amagnetic or optical disk and its corresponding disk drive) as thepermanent storage device 1650.

Other embodiments use a removable storage device (such as a floppy disk,flash drive, or CD-ROM) as the permanent storage device. Like thepermanent storage device 1650, the system memory 1630 is aread-and-write memory device. However, unlike storage device 1650, thesystem memory 1630 is a volatile read-and-write memory, such as a randomaccess memory (RAM). The system memory stores some of the instructionsand data that the processor needs at runtime. In some embodiments, thesets of instructions and/or data used to implement the invention'sprocesses are stored in the system memory 1630, the permanent storagedevice 1650, and/or the read-only memory 1640. For example, the variousmemory units include instructions for processing multimedia items inaccordance with some embodiments.

The bus 1610 also connects to the input devices 1670 and output devices1680. The input devices 1670 enable the user to communicate informationand select commands to the computer system. The input devices includealphanumeric keyboards and pointing devices (also called “cursor controldevices”). The input devices also include audio input devices (e.g.,microphones, MIDI musical instruments, etc.) and video input devices(e.g., video cameras, still cameras, optical scanning devices, etc.).The output devices 1680 include printers, electronic display devicesthat display still or moving images, and electronic audio devices thatplay audio generated by the computer system. For instance, these displaydevices may display a graphical user interface (GUI). The displaydevices include devices such as cathode ray tubes (“CRT”), liquidcrystal displays (“LCD”), plasma display panels (“PDP”),surface-conduction electron-emitter displays (alternatively referred toas a “surface electron display” or “SED”), etc. The audio devicesinclude a PC's sound card and speakers, a speaker on a cellular phone, aBluetooth® earpiece, etc. Some or all of these output devices may bewirelessly or optically connected to the computer system.

Finally, as shown in FIG. 16, bus 1610 also couples computer 1600 to anetwork 1690 through a network adapter (not shown). In this manner, thecomputer can be a part of a network of computers (such as a local areanetwork (“LAN”), a wide area network (“WAN”), an Intranet, or a networkof networks, such as the Internet. For example, the computer 1600 may becoupled to a web server (network 1690) so that a web browser executingon the computer 1600 can interact with the web server as a userinteracts with a GUI that operates in the web browser.

As mentioned above, some embodiments include electronic components, suchas microprocessors, storage and memory that store computer programinstructions in a machine-readable or computer-readable medium(alternatively referred to as computer-readable storage media,machine-readable media, or machine-readable storage media). Someexamples of such computer-readable media include RAM, ROM, read-onlycompact discs (CD-ROM), recordable compact discs (CD-R), rewritablecompact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM,dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g.,DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SDcards, micro-SD cards, etc.), magnetic and/or solid state hard drives,read-only and recordable blu-ray discs, ultra density optical discs, anyother optical or magnetic media, and floppy disks. The computer-readablemedia may store a computer program that is executable by a device suchas an electronics device, a microprocessor, a processor, amulti-processor (e.g., an IC with several processing units on it) andincludes sets of instructions for performing various operations. Thecomputer program excludes any wireless signals, wired download signals,and/or any other ephemeral signals.

Examples of hardware devices configured to store and execute sets ofinstructions include, but are not limited to, ASICs, FPGAs, programmablelogic devices (“PLDs”), ROM, and RAM devices. Examples of computerprograms or computer code include machine code, such as produced by acompiler, and files including higher-level code that are executed by acomputer, an electronic component, or a microprocessor using aninterpreter.

As used in this specification and any claims of this application, theterms “computer”, “computer system”, “server”, “processor”, and “memory”all refer to electronic or other technological devices. These termsexclude people or groups of people. For the purposes of thisspecification, the terms display or displaying mean displaying on anelectronic device. As used in this specification and any claims of thisapplication, the terms “computer readable medium”, “computer readablemedia”, “machine readable medium”, and “machine readable media” areentirely restricted to non-transitory, tangible, physical objects thatstore information in a form that is readable by a computer. These termsexclude any wireless signals, wired download signals, and/or any otherephemeral signals.

It should be recognized by one of ordinary skill in the art that any orall of the components of computer system 1600 may be used in conjunctionwith the invention. Moreover, one of ordinary skill in the art willappreciate that any other system configuration may also be used inconjunction with the invention or components of the invention.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. Moreover, while the examples shownillustrate many individual modules as separate blocks, one of ordinaryskill in the art would recognize that some embodiments may combine thesemodules into a single functional block or element. One of ordinary skillin the art would also recognize that some embodiments may divide aparticular module into multiple modules. Furthermore, specific details(such as details shown in FIGS. 4, 6, 17, 19, etc.) are given as anexample and it is possible to use different circuit implementations toachieve the same results without deviating from the teachings of theinvention.

One of ordinary skill in the art would understand that the invention isnot to be limited by the foregoing illustrative details, but rather isto be defined by the appended claims.

1. A method of providing location-based advertising to a mobile device,the method comprising: receiving content and metadata associated withthe content from the mobile device, the metadata comprising ageographical identification of the content; and sending location-basedinformation to the mobile device based on the geographicalidentification of the content.
 2. The method claim 1, wherein thegeographical identification comprises latitude and longitude coordinatesof a location where the content was generated.
 3. The method of claim 1,wherein the geographical identification comprises an orientation of themobile device when the content was generated, wherein sendinglocation-based information comprises sending information to the mobiledevice based on the orientation of the mobile device.
 4. The method ofclaim 1, wherein the metadata further comprises a time when the contentwas generated, the method further comprising sending time sensitiveinformation to the mobile device.
 5. The method of claim 4, wherein adifference between the time when the content was generated and a currenttime is more than a threshold, the method further comprising:determining a velocity of the mobile device based on a set of previouslyreceived content; determining a direction of the mobile device based onthe set of previously received content; and interpolating the locationof the mobile device based on the location of the generated content andthe velocity and the direction of the mobile device.
 6. The method ofclaim 1 further comprising: determining a context from the contentreceived from the mobile device; and narrowing the location-basedinformation based on the determined context.
 7. The method of claim 1,wherein the location-based information is location-based advertising. 8.The method of claim 1, wherein the location-based information isinformation requested by the mobile device.
 9. The method of claim 1,wherein the mobile device does not transmit live tracking of the mobiledevice location.
 10. The method of claim 1 further comprising sending atelephone link to the mobile device to contact a service providerassociated with the location-based information.
 11. The method of claim10, wherein the telephone link is a link to a Voice over InternetProtocol (VoIP) server to make a call.
 12. The method of claim 1 furthercomprising providing a map associated with the location-basedinformation to the mobile device.
 13. A method of providinglocation-based advertising by a server to a set of mobile devices,wherein the mobile devices exchange collaborative messages with eachother, the method comprising: receiving a message at the server from afirst mobile device, the message comprising a content of a previousmessage sent from a second device to the first device; determining alocation of the second mobile device from the content of the message;and sending location-based advertising to the second mobile device basedon the determined location.
 14. The method of claim 13 furthercomprising narrowing the location-based advertising based on the contextof the message.
 15. The method of claim 13 further comprising:determining a location of the first device based on a content of themessage; and sending location-based advertising to the first mobiledevice based on the determined location.
 16. The method of claim 15further comprising narrowing the location-based advertising sent to thefirst mobile device based on the context of the message.
 17. The methodof claim 15, wherein the message is an email, wherein the locationinformation is embedded in the email.
 18. The method of claim 15,wherein the message is an SMS, wherein the location information isembedded in the SMS.
 19. The method of claim 15, wherein the message isexchanged using a social networking web site, wherein the locationinformation is embedded in the message.
 20. The method of claim 13,wherein the message comprises a content of previous messages from a setof participants exchanging collaborative messages with the first device,the method further comprising: determining a location of at least oneparticipant from the content of the message; and sending location-basedadvertising to a mobile device of the participant based on thedetermined location. 21-35. (canceled)